dpdk book

Since early 1990 network device manufacturer made a lot of innovation in order to increase router speeds. They started from a router node in which everything was computed into the central CPU to reach a situation where the central CPU is less and less used due to a distributed architecture in which lots of action are done in “line cards”.

These progresses have been made thanks to the use of proprietary TCAM (Ternary Content-Addressable Memory) and ASICs (Application-Specific Integrated Circuit) which have been designed to perform table look up and data packets forwarding at high speed.

In early 2000, the Virtualization for x86 computers support has led to lots of innovation into systems domain. Compute virtualization and High-Speed network devices evolution have enabled the Cloud creation.

Later, It appears it was not convenient to manage several isolated network devices each having their own configuration language. Following needs have emerged:

And these desires called for the cloud and SDN technology development.

In Stanford University (US - CA), Clean Slate Research Projects program has been initiated in order to think about how to improve the Internet network architecture. "ETHANE" project was part of this program. Its purpose was to " Design network where connectivity is governed by high-level, global policy". This project is generally known as the first implementation of SDN.

In 2008, a white paper has been proposed by ACM (Association for Computing Machinery) to design a new protocol (OpenFlow) that can program network devices from a network controller.

In 2011, ONF (Open Networking Foundation) has been created to promote SDN Architecture and OpenFlow protocols.

First companies working on SDN have been founded around 2010. (Most of them have now been acquired by main networks or virtualization solution vendors.) In 2007, Martin Casado, who was working on Ethane project has founded Nicira to provide solutions for network virtualization with SDN concept. Nicira has been aquired by vMware in 2012 to develop VMare NSX. In 2016, VMWare also bought PLUMGrid a SDN startup founded in 2013. In 2010, BigSwitch networks has been founded: BigSwitch is proposing a SDN solution. In early 2020, BigSwitch has been acquired by Arista Networks. In 2012, Cisco has created Insieme Networks, a spin-in start-up company working on SDN. In 2013, Cisco take back control on Insieme in order to develop its own SDN solution called ACI (Application Centric Infrastructure). In early 2012, Contrail Systems Inc has been created and aquired at the end of the year by Juniper Networks. In 2013, Alcatel Lucent has created Nuage Networks, a spin-in start-up company working on SDN. Nuage Networks is now an affiliate of Nokia.

The road of SDN development and its history is never straighforward and looks more nuanced than a single storyline might suggest. It’s actually far more complex to be described in a short section. This diagram from [sdn-history] shows developments in programmable networking over the past 20 years, and their chronological relationship to advances in network virtualization.

The concept of SDN, and the term itself, are both very broad and often confusing. There is no real accurate definition of SDN, and vendors usually take it very differently. Initially it was used to in Stanford’s OpenFlow project, and later it has been extended to include a much wider area of technologies. Discussion about each vendor’s exact SDN definition is beyond the scope of this book. But we generally consider that a SDN solution has to provide one to several of following characteristics:

According to [onf-sdn-definition], Software-Defined Networking (SDN) is:

In this diagram, you can see that SDN allows simple high-level policies in the "application layer" to modify the network, because the device level dependency is eliminated to some extent. The network administrator can operate the different vendor-specific devices in the "infrastructure layer" from a single software console - the "control layer". The "controller" in control layer is designed with such a way that it can view the whole network globally. This controller design helps a lot to introduce functionalities or programs as they just needs to talk to the centralized controller, without the need to know all details communicating with each individual device. These details are hidden by the controller from the applications.

Several expectations are behind this new model:

traditionally, A typical network device (e.g. a router) has following planes:

configuration and control plane are located in device’s main processor card, oftenly called "routing engine", or "routing switching engine". The forwarding plane is located in the device’s packet forwarding card, oftenly called "line card".

SDN architecture is built with 3 layers:

In a traditional infrastructure, the route calculation is made on each individual router. each router needs to run one or several routing protocols, through which it exchanges routes with the rest routers in the network, and eventually, based on the route information learned, each router assumes it gains enough knowledge about the network in order to make the forwarding decision. From the network perspective, the control plane is distributed in each individual router, and the end to end routing path is the result of all decisions made by the control plane located on each router.

The control plane on one router may look like this:

In reality, for example, a simplified Juniper MX control plane typical looks like this:

Running a control plane on each router make it very hard to manage, because each individual network device needs to be carefully configured. It requires extensive, vendor-specific experiences and skills to configure the device. The high number of configuration points often make it very challenging to build a robust network. Flexibility is also a recurring hurdle for traditional networks since most routers run proprietary hardware and software.

In contrast, in SDN networking, Control and Configuration functions are gathered into a "SDN controller" which is controlling Network devices. The new architecture intends to provide a completely new way to configure the network. This new Cloud infrastructure brings:

Let’s compare the two architectures:

This SDN infrastructure uses a centralized configuration and control point. route calculation is done centrally in the controller and distributed into each SDN network node. Well the idea looks good and simple, it requires a few foundamental protocols and infrastructures to be implemented before this model can work:

This underlay network infrastructure is playing the same role that the local switch fabric is doing inside a standalone router between the control processor card and lines cards. Based on it, A "overlay" network can be built by the controller, which basically hides underlay network infrastructure details from the applications so they will focus on the high level service implementations. we’ll talk more about "underlay" and "overlay" in the next section.

convenient as it is, this makes the controller the weakest point in the whole model. Think of what will happen if this SDN controller, serving as the "brain", stops working. Everything will be frozen and nothing works as expected, or even worse, some part of the infrastructure continues to run but in an unexpected way, which will very likely trigger bigger issues to other part of the network.

Lots of efforts are done by each SDN solution supplier to solve this weakness. A common and efficient practice is to use clustered architecture to build a highly resilient controller cluster. e.g 3 SDN controllers can load balance and/or backup each other. on failure of one or two, the other one can still make the whole cluster survive, giving the operator longer maintanence windows to fix the problem.

In SDN architecture, each network node is connected to a physical network infrastructure. This physical network which is providing basic connectivity between network nodes is called the "underlay" network infrastructure. sometimes it is also called "fabric", and typically it’s a plane L3 IP network.

very often The underlay needs to separate between different administrative domains (often called "tenants"), switch within the same L2 broadcast domain, route between L2 broadcast domains, provide IP separation via VRFs, and etc. This is implemented in the form of "overlay" networks. The overlay network is a logical network that runs on top of the underlay network. The overlay is formed of tunnels to carry the traffic across the L3 fabric.

Today the industry began to shift in the direction of building L3 data centers and L3 infrastructures, mostly due to the rich features coming from L3 technologies, e.g, ECMP load balancing, flooding control, etc. But the L2 traffic does not disappear and most likely it never will. There are always the desire that a group of network users need to reside in the same L2 network - typically a VLAN. However, In today’s virtualization environment, a user’s VM can be spawned in any compute located anywhere in the L3 cluster. Even if 2 VMs are spawned in the same server, there is often a need to move them around between different servers without changing their networking attributes. These requirements to make a VM always belonging to the "same VLAN" calls for an overlay model over the L3 network. In other words, we need a new mechanism to allow us to tunnel L2 Ethernet domains with different encapsulations over an L3 network.

For example, in SDN node1 we were running VM11 and VM12, they were both serving same sales department and so they were located in same VLAN. because of some administrative requirement, VM12 needs to be moved to another physical SDN node2 which, may be physically located in another rack that is a few router "hops" away. Now we need to ensure not only data packet from VM11 in SDN node1 to be able to reach VM12 in SDN node2, but also they are talking to each other as if they are still in the same VLAN, exactly the same way as before just as if VM12 has never moved. This ability to make the "local" (in same VLAN) traffic to traverse transparently across underlay network infrastructure calls for a packet encapsulation, or "tunneling" mechanism in SDN networks.

Indeed, without such an encapsulation mechanism, traditional segmentation solutions (VLAN, VRF) would have to be provided by the physical infrastructure and implemented up to each SDN node, in order to provide an isolated transportation channel for each customer network connected to the SDN infrastructure.

Encapsulation protocols used in SDN networks have to provide:

Several encapsulation protocols are used into SDN networks:

These encapsulation protocols are providing Overlay connectivity which is required between customers workload connected to the SDN infrastructure. Each SDN node is call a VTEP (Virtual Tunnel End Point) as it is starting and terminating the overlay tunnels.

We’ve seen "openflow" marked as one of the possible interfaces in the "SDN layer" section. Now we’ll introduce the concept of "southbound" and "northbound" interface and other available choices in today’s industry.

The "southbound" interface resides between the controller in "control layer" and network devices in "infrastructure layer". Basically what it does is to provide a means of communication between the 2 layers. Based on the demands and needs, a SDN Controller will dynamically changes the configuration or routing information of network devices. For example, a new VM will advertise a new subnet or host routes when it is spawned in a server, this advertisement will be delivered to SDN controller via a southbound protocol. Accordingly, SDN controller collects all routing updates from the whole SDN cluster through the southbound interfaces, and decides the most current and best route entries, then, it may "reflect" these information to all other network devices or VMs. This ensures all devices having the most uptodate routing information in real time. Among others, examples of the most well-known southbound interfaces in the industry are openflow, OVSDB and XMPP.

openflow is a protocol that sends flow information into the virtual switch so the switch can forward the packets between the different ports. Flows are defined based on different criteria such as traffic between a source MAC address and a destination MAC address, source and destination IP addresses, TCP ports, VLANs, tunnels, and so on.

OpenFlow is one of the most widely deployed southbound standard from open source community. It first made its appearance in 2008 by Martin Casado at Stanford University. The appearance of OpenFlow was one of the main factors which gave birth to Software Defined Networking.

OpenFlow provides various information for the Controller. It generates the event-based messages in case of port or link changes. The protocol generates a flow based statistic for the forwarding network device and passes it to the controller.

OpenFlow also provides a rich set of protocol specifications for effective communication at the controller and switching element side. Open Flow provides an open source platform for Research Community.

Every physical or virtual OpenFlow-enabled network (data plane) devices in the SDN domain needs to first register with the OpenFlow controller. The registration process is completed via an OpenFlow HELLO packet originating from the OpenFlow device to the SDN controller.

abbreviation for "Open vSwitch Database". unlike openflow, OVSDB is a southbound API designed to provide additional management or configuration capabilities like networking functions. With OVSDB we can create the virtual switch instances, set the interfaces and connect them to the switches. We can also provide the QoS policy for the interfaces. OVSDB is a protocol written in the JavaScript Object Notation (JSON) that basically sends and receives commands via JSON RPCs.

The northbound interface provides connectivity between the controller and the network applications running in management plane. As we already discussed that southbound interface has OpenFlow as open source protocol, northbound lacks such type of protocol standards. However with the advancement of technology now we have a wide range of northbound API support like ad-hoc API’s, RESTful APIs etc. The selection of northbound interface usually depends on the programming language used in application development.

with the development of virtualization, SDN technologies and their ecology in recent years, more and more terms and changing of these terms emerge in the networking industry. a lot of confusions have rised, often because of terms are referring different things when they are used in different context. Sometimes the latest term the industry uses is a particular technology such as VNF or a concept such as NFV. Terms rise and fall out of favor as the industry evolves. In recent years the terms such as openstack, NVF/VNF has become the industry’s favorite buzzword. This raises the question - just what is openstack, NVF/VNF and what are the relationships with SDN?

NFV/VNF sounds like new buzzwords, but those technologies have been around for years. according to ETSI:

NFV means "network function virtualization", it stands for an "operation framework for orchestrating and automating VNFs". And VNF means "virtualized network function", such as virtualized routers, firewalls, load balancers, traffic optimizers, IDS or IPS, web application protectors, and so on.

in a nut shell you can think of NVF as a "concept", or "framework" to virtualize certain network functions, while VNF is the implementations of each individual network functions. Among others, firewalls and load balancers are the two most common VNFs in the industry, especially for deployments inside data centers. When you read today’s documents about virtualization technology, you will see the terms in such a pattern like "vXX" (e.g. vSRX, vMX), or "cXX" (e.g. cSRX) very often. that letter v indicates it is a "virtualized" product, while letter c - "containerized" is its container version.

Jointly launched by NASA and Rackspace in 2012, Openstack has rapidly gained popularity in many enterprise data centres. It is one of the most used open source cloud computing platform to support software development and Big Data analytics. OpenStack comprises a set of software modules, e.g, compute, storage & networking modules, which works together to provide an open source choice for building private & public cloud environments. As an IaaS (Infrastructure As A Service) open source implementation ,it provides a wide range of services, from basic service like computing service, storage service, networking service, etc, to advanced services like database, container orchestration and others.

You can think of Openstack as an abstraction layer providing a cloud environment on your promise. with openstack installed in your servers, ,you can spawn a VM, consume and recycle it when you are done, all in seconds. under that abstraction layer, Openstack hides most complexities of automation and orchestration of diverse underlying resources like compute, storage and networking. you could choose Servers, storage, networking devices from your favorite vendors to build the underlying infrastructure, and openstack will "consume" all of them and expose to the user as a pool of common "resources": number of CPUs, RAMs, hard disk spaces, IP addresses, etc. The user does not (need to) care about vendor and brand details.

If we compare openstack with SDN, it’s not hard to see that the two model shares some common features. Both models provide certain level of abstractions, hide the low level hardware details and expose to upper level user applications. the differences are somewhat subtle to describe in just a few words. First off, although there are various distributions from different vendors, they share common core components that is managed by the OpenStack Foundation. SDN is more of a "framework" or an "approach" to manage the network dynamically, which can be implemented with totally different software techniques. Secondly, From the perspective of technical ecological coverage, the ecological aspects of OpenStack are much wider, because networking is just one of its services that is implemented by its Neutron component among it’s other various plugins. SDN, and its ecology, in contrast, mainly focus on the networking. There are also difference in the way that Neutron works comparing with how a typical SDN controller works. OpenStack Neutron focuses on providing network services for virtual machines, containers, physical servers, etc, and provides a unified northbound REST API to users, SDN focuses on configuration and management of forwarding control toward the underlaying network device, it not only provides user-oriented northbound API, but also provides standard southbound API to communicating with various hardware devices.

As we’ve mentioned in previous sections, SDN is a networking scenario which changes the traditional network architecture by bringing all control functionalities to a single location and making centralized decisions. SDN controllers are the brain of SDN architecture, which perform the control decision tasks while routing the packets. Centralized decision capability for routing enhances the network performance. As a result, SDN controller is the core components of any SDN solutions.

While working with SDN architecture, one of the major point of concerns is which controller and solution should be selected for deployment. There are quite a few SDN controller and solutions implementations from various vendors, and every solution has its own pros and cons along with its working domain. In this section we’ll review some of the popular SDN controllers in the market, and the corresponding SDN solutions.

OpenDaylight, aften abbreviated as ODL, is a Java based open source project started from 2013, it was originally led by IBM and Cisco but later hosted under the Linux Foundation. it was the first open source Controller that can support non-OpenFlow southbound protocols, which can make it much easier to be integrated with multiple vendors.

ODL is a modular platform for SDN. It is not a single piece of software. It is a modular platform for integrating multiple plugins and modules under one umbrella There are many plugins and modules built for OpenDaylight. Some are in production, while some are still under development.

Some of the initial SDN controllers had their southbound APIs tightly bound to OpenFlow, But as we can see from the diagram, besides openflow, many other southbound protocols that are available in today’s market are also supported. Examples are NETCONF, OVSDB, SNMP, BGP, etc. Support of these protocols are done in a modular method in the form of different plugins, which are linked dynamically to a central component named "Service Abstraction Layer (SAL)". SAL does translations between the SDN application and the underlaying network equipments. for instance, when it receives a service request from a SDN application, typically via high level API calls (northbound), it understands the API call and translates the request to a language that the underlying network equipments can also understand. That language is one of the southbound protocols.

While this "translation" is transparent to the SDN application, ODL itself needs to know all the details about how to talk to each one of the network devices it supports, their features, capabilities etc. a topology manager module in OLD manages this type of information. What topology manager does is to collect topology related information from various modules and protocols, such as ARP, host tracker, device manager, switch manager, OpenFlow, etc, and based on these info, it visualize the network topology by drawing a diagram dynamically, all the managed devices and how they are connected together will be showed in it.

any topology changes, such as adding new devices, will be updated in the database and reflected immediately in the diagram.

Remember earlier we mentioned that an SDN controller has "global view" of the whole SDN network. In that sense ODL has all necessary visibility and knowledge of the network that can be used to draw the network diagram in realtime.

OVS is one of the most popular and "production quality" open source implementation of a multilayer virtual switch. OVS was created by Nicira back in 2009, which was acquired by VMware. It is licensed under the Apache 2.0 license and provided by Linux Foundation.The virtual switch basically does most of the jobs you could expect a physical switch does, but in a software method. OVS is typically running with linux hypervisors like KVM and can be loaded on a Linux kernel. OVS supports most features supported in tradtional physical switches, such as:

Beside functions of traditional switches, the bigger advantage of OVS is that it also has native support to SDN solution via OVSDB and OpenFlow protocols. That means any SDN controller can integrate OVS via these 2 open standard protocols. Therefore OVS can work either as a standalone L2 switch within a hypervisor host, or it can be managed and programmed via an SDN controller, such as ODL. that is why it is used in so many open source and commercial virtualization projects.

Open vSwitch introduced an architecture that comprises an SDN controller that configures and manages virtual switches via the OVSDB protocol and pushes flows inside the switches via the OpenFlow protocol. In general the OVS comprises the following components:

The architecture is described in this figure:

This is a configuration database that controls and stores the switch-level configuration. It contains information on creating bridges, attaching interfaces, attaching tunnels, and so on. these objects are organized in the form of a set of different tables that points to each other in a certain sequence:

For example, an entry in the top level OVS table points to a brige table, which has items pointing to a port table, which in turn, points to a interface table. The stateful database make sure the system can recover back to the exact status it was in case it rebooted. The ovsdb-server database talks to the outside controller via the OVSDB protocol.

This is the heart of OVS and where flow processing happens. ovsdb-vswitchd has all the information (e.g. bridges, flow tables, etc) needed to forward packets. It has different interfaces to other components. Inside of the hypervisor, it connects to ovsdb-server via the OVSDB protocol, and to the kernel module via a Linux Netlink interface. To outside controller, it runs OpenFlow protocol to exchange flow information.

Ovsdb-vswitchd pushes the flows to the kernel module for fast forwarding. When the first packet arrives, it goes through the kernel module, where the headers are hashed to find a flow entry. If the flow entry is not found, the packet goes to ovsdb-vswitchd for normal processing. Ovsdb-vswitchd then pushes the flow to be cached inside the module kernel. If a similar flow comes in, it is forwarded via the fast path inside the kernel module. The kernel module does not contain any of the OpenFlow tables that are known to ovsdb-vswitchd; rather, it contains the result of the different lookups in the flow tables. The kernel module also handles the tunneling of packets via protocols such as GRE, VXLAN, and others.

quote from calico official website:

Calico has been an open-source project from day one. It was originally designed for today’s modern cloud-native world and runs on both public and private clouds. Its reputation mostly comes from it’s deplayment in Kubernetes and its ecosystem environments. Today Calico has become one of the most popularly used kubernetes CNIes and many enterprises using it at scale.

Comparing with other overlay network SDN solutions, Calico is special in the sense that it does not use any overlay networking design or tunneling protocols, nor does it require NAT. Instead it uses a plain IP networking fabric to enables host to host and pod to pod networking. The basic idea is to provides Layer 3 networking capabilities and associates a virtual router with each node, so that each node is behaving like a traditional router, or a "virtual router". We know that a typical Internet router relies on routing protocols like OSPF, BGP to learn and advertise the routing information, and That is the way a node in calico networking works. It chooses BGP, because of it’s simple, industry’s current best practice, and the only protocol that sufficiently scale.

calico uses a policy engine to deliver high-level network policy management.

Calico is made up of the following components:

This is calico "agent" - a daemon that runs on every workload, for example on nodes that host containers or VMs. it is the one that performs most of the "magics" in the calico stack. It is responsible for programming routes and ACLs, and anything else required on the host, in order to provide the desired connectivity for the endpoints on that host.

Depending on the specific orchestrator environment, Felix is responsible for the following tasks:

It does all this by connecting to etcd and reading information from there. It runs inside the calico/node DaemonSet along with confd and BIRD.

The orchestrator plugins are essentially responsible for API translations. Calico has a separate plugin for each major cloud orchestration platforms (e.g. OpenStack, Kubernetes).

For example in openstack environment, a Calico Neutron ML2 driver integrates with Neutron’s ML2 plugin to allows users to configure the Calico network simply by making Neutron API calls. This provides seamless integration with Neutron.

the backend data store for all the information Calico needs. it can be the same of different etcd that kubernetes use. it has at least, but not limited to the following information: * list of all workloads (endpoints) * BGP configuration * policys from user (e.g. defined via the calicoctl tool) * information about each container (pod name, IP, etc), received from calico CNI

Calico makes uses of BGP to propagate routes between hosts. And the BGP "speaker" in calico is BIRD - a routing daemon that runs on every host that also hosts Felix module in the Kubernetes cluster, usually as a DaemonSet. It ’s included in the calico/node container. it’s role is to read routing state that Felix programs into the kernel and distribute it around the data center. comparing with what Felix does, one of the main differences is that Felix "insert" routes into the linux kernel FIB and BIRD "distribute" them to all other nodes in the deployment, this turns each host to a virtual Internet BGP router ("vRouter"), and ensures that traffic is efficiently routed around the deployment.

confd is a simple configuration management tool. In Calico, BIRD does not deal with etcd directly, it is another module "confd" that reads the BGP configuration from etcd and feed to BIRD in the form of configurations files in disk.

configure IP, routes CNI stands for "container networking interface".

There’s an interface for each pod, When the container spun up, calico (via CNI) created an interface for us and assigned it to the pod.

when a new pod starts up, Calico will: - query the kubernetes API to determine the pod exists and that it’s on this node - assigns the pod an IP address from within its IPAM - create an interface on the host so that the container can get an address - tell the kubernetes API about this new IP

as the name indicated already, Calico’s IPAM plugin is responsible for "IP address management". when a new container is spawn, calico IPAM plugin reads information from etcd database to decide which IP is available to be allocated to the container. the IP address by default will be allocated in the unit of /26 "block". a block is essentially a subnet which aggregate the routes to save routing table spaces.

The Virtualized Cloud Platform (VCP) is created by Nuage networks. It provides a "policy-based" SDN platform that has a data plane built on top of the open source OVS, and a SDN controller built on open standards.

The Nuage platform uses overlays to provide policy-based networking between different clouding environment (Kubernetes Pods or non-Kubernetes environments such as VMs and bare metal servers). it also has a real-time analytics engine to monitor Kubernetes applications.

All components can be installed in containers. There are no special hardware requirements.

In Nuage VCP, The Virtualised Services Directory (VSD) is a policy engine, business logic and analytics engine that supports the abstract definition of network services. Through RESTful APIs to VSD, administrators can define and refine service designs and incorporate enterprise policies.

It is a web-based, graphical console that connects to all of the VRS nodes in the network to manage their deployment and configuration.

The VSD policy & analytics engine presents a unified web interface where configuration and monitoring data is presented. The VSD is API-enabled for integration with other orchestration tools. Alternatively, you can develop your apps. Either way, the VSD is based on tools from the service provider world, and therefore scaling potential looks very good. It integrates multiple data centre networks by linking VSDs together and exchanging policy data.

Nuage Virtual Services Controllers (VSC) works between VSD and VRS. policies from VSD is distributed through a number of VSC to all of the VRS nodes in the network to manage their deployment and configuration.

VSC is SDN controller in Nuage VCP architecture. it provides a robust control plane for the datacenter network, maintaining a full per-tenant view of network and service topologies. Through network APIs that use southbound interfaces (e.g. OpenFlow), VSC programs the datacenter network independent of different hardwares.

The VSC implements an OSPF, IS-IS or BGP listener to monitor the state of the physical network. Therefore, if routes starts flapping, the VSC is able to incorporate those events into the decision tree.

while scalability in a single data center can be achieved by setting up multiple VSC, each handling a certain group of VRS devices, scalability between multiple data centres can be achieved by connecting VSC controllers horizontally at the top of the hierarchy.

As shown in the diagram above, VSC controllers are synchronised using MP-BGP. A BGP connection peers with PE routers at the WAN edge, and then the VSC controller uses MP-BGP to synchronise controller state & configuration with VSCs in other data centres. This is vital for end-to-end network stability.

When dVRS devices are communicating to non-local dVRS devices, data is tunnelled in MPLS-over-GRE to the PE router.

The VRS module serves as a virtual endpoint for network services. It detects changes in the compute environment as they occur and instantaneously triggers policy-based responses to ensure that the network connectivity needs of applications are met.

configuration of the VRS is derived from a series of templates.

Each VRS routes traffic into the network according to its flow table. Therefore, the entire VRS system performs routing at the edge of the network.

A VRS can’t make a forwarding decision in a vacuum, as events in the underlying physical network must be considered. Nuage Networks has extensively considered how to provide the VSC controller with all the information required to have a complete model of the network.

The Tungsten Fabric (TF), is an open-standard based, proactive overlay SDN solution. It works with existing physical network devices and help address the networking challenges for self-service, automated, and vertically integrated cloud architecture. It also improves scalability through a proactive overlay virtual network technique.

TF controller integrates with most of the popular cloud management systems such as OpenStack, vmware, and Kubernetes. TF’s focus is to provide networking connectivity and functionalities, and enforce user-defined network and security policies to the various of workloads based on different platforms and orchestrators.

Tungsten Fabric’s primary claim to fame is that it is diligently multi-cloud and multi-stack. Today it supports:

TF fits seamlessly into LFN (Linux Foundation Networking) mission to foster open source innovation in the networking space.

The TF system is implemented as a set of nodes running on general-purpose x86 servers. Each node can be implemented as a separate physical server, or VM.

Initially, "Contrail" was a product of a startup company "Contrail system", which was acquired by Juniper Networks in Dec. 2012. It was open sourced in 2013 with a new name "OpenContrail" under the Apache 2.0 license, which means that anyone can use and modify the code of "Opencontrail" system without any obligation to publish or release the modifications. In early 2018, it was rebranded to "Tungsten Fabric" (abbreviated as "TF") as it transitioned into a fully-fledged Linux Foundation project. currently TF is still managed by the Linux Foundation.

Juniper also maintains a commercial version of the Contrail system, and provides commercial support to the payed users. Both The open-source version and commerical version of the Contrail system provide the same full functionalities, features and performances.

TF consists of two main components:

The communication between the controller and vRouters is via XMPP, which is a widely used messaging protocol.

A high level Tungsten Fabric architecture is shown below:

Kernel-based Virtual Machine (KVM) is an open source virtualization technology built into Linux. It provides hardware assist to the virtualization software, using built-in CPU virtualization technology to reduce virtualization overheads (cache, I/O, memory) and improving security.

QEMU is a hosted virtual machine emulator that provides a set of different hardware and device models for the guest machine. For the host, QEMU appears as a regular process scheduled by the standard Linux scheduler, with its own process memory. In the process, QEMU allocates a memory region that the guest sees as physical and executes the virtual machine’s CPU instructions.

With KVM, QEMU can just create a virtual machine with virtual CPUs (vCPUs) that the processor is aware of and runs native-speed instructions. When a special instruction is reached by KVM, like the ones that interacts with the devices or to special memory regions, vCPU pauses and informs QEMU of the cause of pause, allowing hypervisor to react to that event.

LibVirt is an Open Source toolkit to manage virtualization platforms. Libvirt is collection of softwares which allow to manage virtual machines and other virtualization functionality, such as storage and network interface management. LibVirt is proposing to define virtual components in a XML-formatted configurations, that are able to be translated into QEMU command line.

Inter process communication (IPC) is a mechanism which allows processes to communicate with each other and synchronize their actions. The communication between these processes can be considered as a method of cooperation between them.

IPC is used in network virtualization in order to be able to exchange data between different distributed processes of a same application (for example, virtio frontend and backend, contrail vrouter agent and dataplane, etc …​) or between processes of distinct applications (e.g., contrail vrouter and QEMU virtio, virtio and VFIO, and so on)

Two different modes of communication are used for IPC:

Like CPU virtualization, two kinds of methods are used to virtualize network devices:

Software Based Emulation are widely supported but can suffer of poor performance. Hardware assisted Emulation if providing good performance thanks to hardware acceleration, but it requires to use a hardware that supports some specific features.

The following two emulated network devices are provided with QEMU/KVM:

Virtio is an open specification for virtual machines' data I/O communication, offering a straightforward, efficient, standard and extensible mechanism for virtual devices, rather than boutique per-environment or per-OS mechanisms. It uses the fact that the guest can share memory with the host for I/O to implement that.

Virtio was developed as a standardized open interface for virtual machines (VMs) to access simplified devices such as block devices and network adaptors.

KVM guests usually have access to software based emulated NIC device (either para-virtualized devices with virtio or traditional emulated devices). On host machines which have Intel VT-d or AMD IOMMU hardware support, another option is possible. PCI devices may be assigned directly to the guest, allowing the device to be used with minimal performance overhead.

Assigned devices are physical devices that are exposed to the virtual machine. This method is also known as passthrough.

The VT-d or AMD IOMMU extensions must be enabled in BIOS in order to be able to perform for device Direct Assignment:

Two methods are supported:

With VFIO the Physical device is exposed to the host user space memory and is made visible from the guest VM it has been assigned.

Single Root I/O Virtualization (SR-IOV) specification is defined by the PCI-SIG (PCI Special Interest Group). This is a PCI Express (PCI-e) that extends a single physical PCI function to share its PCI resources as separate virtual functions (VFs).

The physical function contains the SR-IOV capability structure and manages the SR-IOV functionality (it can be used to configure and control a PCIe device).

A single physical port (root port) presents multiple, separate virtual devices as unique PCI device functions (up to 256 virtual functions – depends on device capabilities).

Each virtual device may have its own unique PCI configuration space, memory-mapped registers, and individual MSI-based interrupts. Unlike a physical function, a virtual function can only configure its own behavior. Each virtual function can be directly connected to a virtual machine via PCI device assignment (passthrough mode).

SR-IOV improves network device performance for each virtual machine as it can share a single physical device between several virtual machines using device direct I/O assignment method.

With SR-IOV, each VM has a direct access to the physical network using the assigned virtual function interface allocated to each. They can communicate altogether using the Virtual Ethernet Bridge provided by the NIC card. A virtual switch can also use SRIOV to get access to the physical network. VM using SRIOV assigned virtual function device has a direct access to the physical network and are not connected to any intermediate virtual network switch or router.

Following command can be used to check whether SR-IOV is supported or not on a physical NIC card:

$ lspci -s <NIC_BDF> -vvv | grep -i "Single Root I/O Virtualization"

VirtIO is bringing lots of flexibility. VirtIO is offering a standardized driver which is fully independent of the hardware used on the physical platform hosting VM instances.

When virtio connectivity is used VM can be easily migrated from one host to another using "live migration" feature. When SRIOV is use, this live migration is not an easy task and is not really possible to achieve.

Indeed, network driver used by VM depends on used hardware on the bare metal node which are hosting them. In order to make VM migration from one bare metal node to another, both nodes must at least to use same hardware NIC model. But when SRIOV is used VM connectivity is having barely the same performance has a real physical NIC, whereas with VirtIO, performance could be poor.

Also, SRIOV, providing a direct access to the physical NIC is making host virtual network nodes (virtual router/switch) used by SDN solution totally blind about VM using such connectivity. Local traffic switching between VM connected on a same SRIOV physical card is achieve by the Virtual Ethernet bridge proposed by SRIOV. Communication between VM connected onto distinct SRIOV physical ports must rely on physical network.

SDN vswitch/vrouter usage is very limited when SRIOV is used. Indeed, packet switching between VMs which are using VFs of a same SR-IOV physical port are using the physical Virtual Ethernet Bridge hosted in the physical NIC.

Only some few use cases are relevant, which are:

With SRIOV we are getting high performance but with poor flexibility and no network virtualization features. With VirtIO we are getting a high level of network virtualization suitable for SDN, which is very flexible with poor performances.

For SDN use cases, we need network virtualization features and performance. DPDK will bring both.

DPDK is dedicating one (or more) CPU to one (or more) thread that are continuously polling a one (or more) DPDK NIC RX queue. CPU on which a DPDK polling thread is started will be loaded at 100% whatever there some packets to process or not, as no interrupt mechanism is used in DPDK to warn the DPDK application that a packet has been received.

Using DPDK library API, physical NIC packets will be made available into user space memory in which the DPDK application is running. So, when DPDK is used there is no user space to kernel space context switching and it saves lots of CPU cycles. Also, the host memory is using large continuous memory area, the huge pages, which allow large data transfers and avoid high data fragmentation in memory which would require a higher memory management effort at the application level. Such a fragmentation would also cost some precious CPU cycles.

Hence, most of the CPU cycles of DPDK pinned CPU are used for polling and processing packets delivered by the physical NIC in DPDK queues. As a result, the packet forwarding task can be processed at a very high speed. If one CPU is not powerful enough to manage incoming packets that are hitting the physical NIC at a very high rate; we can allocate an additional one to the DPDK application in order to increase its packet processing capacity.

A DPDK application is a multi-thread program that is using DPDK library to process network data. In order to scale, we can start several packet polling and processing threads (each one pinned on a dedicated CPU) that are running in parallel.

3 main components are involved into a DPDK application:

Data Plane Development Kit (DPDK) is a set of data plane libraries and network interface controller drivers for fast packet processing, currently managed as an open-source project under the Linux Foundation.

The main goal of the DPDK is to provide a simple, complete framework for fast packet processing in data plane applications.

The framework creates a set of libraries for specific environments through the creation of an Environment Abstraction Layer (EAL), which may be specific to a mode of the Intel® architecture (32-bit or 64-bit), Linux* user space compilers or a specific platform.

These environments are created through the use of make files and configuration files. Once the EAL library is created, the user may link with the library to create their own applications.

The DPDK implements a "run to completion model" for packet processing, where all resources must be allocated prior to calling Data Plane applications, running as execution units on logical processing cores.

The model does not support a scheduler and all devices are accessed by polling. The primary reason for not using interrupts is the performance overhead imposed by interrupt processing.

For more information please refer to dpdk.org documents http://dpdk.org/doc/guides/prog_guide/index.html

DPDK is a set of programing libraries that can be used to create an application that needs to process network packets at a high speed. DPDK is proposing following functions:

In order to reduce Linux user to kernel space context switching all these functions are made available by DPDK into the user space where applications are running. User applications using DPDK libraries have a direct access to the NIC cards, without passing through a NIC Kernel driver as it is required when DPDK is not used.

DPDK is allowing to build user-space multi-thread network application using the POSIX thread (pthread) library.

DPDK is a framework which is made of several libraries:

Ethdev library exposes APIs to use the networking functions of DPDK NIC devices. The bottom half part of ethdev is implemented by NIC PMD drivers. Thus some features may not be implemented.

Poll Mode ethernet Drivers (PMDs) are a key component for DPDK. These PMDs by-pass the kernel and are providing a direct access to the Network Interface Cards (NIC) used with DPDK.

Linux user space device enablers (UIO or VFIO) are provided by Linux Kernel and are required to run DPDK. They are allowing to discover and expose PCI devices information and address space through the /sys directory tree.

DPDK libraries are allowing kernel-bypass application development:

The diagram below is providing an overview of DPDK libraries.

Only few libraries have been described in this diagram: Set of libraries is enriched at each new DPDK release (cf: https://www.dpdk.org/).

The Environment Abstraction Layer (EAL) is responsible to provide access to low-level resources such as hardware and memory space. It provides a generic interface that hides the environment specifics from the applications and libraries. The EAL performs physical memory allocation using mmap() in hugetlbfs (using huge page sizes to increase performance).

Provided services by EAL are:

DPDK has a highly optimized memory manager. DPDK works on a group of fixed size objects called a mempool. Every one of them are pre-allocated. DPDK does not encourage dynamic allocations because it consumes a lot of CPU cycles and it is a speed killer.

DPDK stores incoming packets into mbufs (memory buffers). DPDK pre-allocates a set of mbufs and keeps it in a pool called mempool.

DPDK makes use of mempools each time it needs to allocate a mbuf where packets are stored. Instead of allocating a single mbuf, DPDK do a bulk allocation, or bulk free once packets are consumed. By doing this, packets to be processed (mbufs) are already in cache memory. Therefore, DPDK is very cache friendly.

Mempool has further optimizations. It is very cache friendly. Everything is aligned to the cache and has a some mbufs allocated for each DPDK thread or lcore. Each mempool are also bound with rings which are referencing mbufs containing packets stored into mempool.

Each ring is a highly optimized lockless ring. It can be used by several lcores in a multi-producer/multi-consumer kind of scenario without locks. By avoiding locks, DPDK gets large performance gains, as data structures locking is also a speed killer.

Network Data are stored in compute central memory (in huge page area).

DPDK uses message buffers known as mbufs to store packet data into the host memory. These mbufs are stored in memory pools known as mempools.

mbufs are storing DPDK NIC incoming and outgoing packets which have to be processed by the DPDK application.

DPDK queues are not storing the packets but a pointer onto the real packet. It avoids performing a data transfer that would be needed when packets have to be forward from a DPDK NIC to another.

Packets are not moved from one queue to another, but these are descriptors (pointers) that are moving from one queue to another.

Descriptors are set up as a ring. A ring is a circular array of descriptors. Each ring describes a single direction DPDK NIC queue. Each DPDK NIC queue is made up of 2 rings (1 per direction: 1 RX ring, 1 TX ring).

Each descriptor points onto a packet that has been received (RX ring) or that is going to be transmitted (TX ring).

The more descriptors RX/TX rings are containing, the more memory size will be required in each mempool (number of mbufs) to store data.

DPDK application is only processing packets that are exposed in user space host OS memory.
DPDK rings are an abstraction of the real NIC queues: DPDK is using DMA to keep synchronized at anytime between the NIC hardware queues and its DPDK representation in the host memory.

Multithreading is the ability of a CPU (single core in a multi-core processor architecture) to provide multiple threads of execution concurrent. In a multithreaded application, the threads share some CPU resources memory:

A single Linux process can contain multiple threads, all of which are executing the same program. These threads share the same global memory (data and heap segments), but each thread has its own stack (local variables).

Linux pThreads (POSIX threads) is a C library which contains a set functions that are allowing to manage threads into an application. DPDK is using Linux pThreads library.

DPDK is using threads that are designed as "lcore”. A “lcore" refers to an EAL thread, which is really a Linux pthread, which is running onto a single processor execution unit.

Lcores are not sharing CPU units. Nevertheless, if the host processor supports hyperthreading, a core may include several lcores or threads.

lcores are used to run DPDK application packet processing threads. Several packet processing models are proposed by DPDK. The simplest one is the Run-To-Completion model.

Run-to-Completion, is using a single thread (lcore) for end to end packet processing (packet polling, processing and forwarding).

A complex application is typically split across multiple cores, with cores communicating through Software queues.

Packet Framework facilitates the creation of pipelines. Each pipeling thread is assigned to a CPU and is using software queues like output or/and input ports.

For instance, Contrail DPDK vRouter is using such a model for GRE encapsulated packet processing.

It is possible to create Control Threads. Those threads can be used for management/infrastructure tasks and are used internally by DPDK for multi process support and interrupt handling.

DPDK service cores enables a dynamic way of performing work on DPDK lcores. Service core support is built into the EAL, and an API is provided to optionally allow applications to control how the service cores are used at runtime.

PCI devices have a set of registers referred to as configuration space for devices. These configuration space registers are mapped to host memory locations.

When a PCI device is enabled, the system’s device drivers (by writing configuration commands to the PCI controller) programs the Base Address Registers (BAR) to inform the PCI device of its address mapping. Next, the host operating system is able to address this PCI device.

With usual Linux NIC Kernel, both NIC configuration and Packet processing is done in Kernel Space. User applications which have to establish a TCP connection or send a UDP packet is using the sockets API, exposed by libc library.

Linux Packet Processing with sockets API is requiring following operations which are costly:

With usual Linux Drivers most of operations are occurring in Kernel modes and are requiring lots of user space to kernel space context switching and interruption mechanisms. The heavy context switching usage is costing lots of CPU cycles and is a limiting the numbers of packets that a CPU is able to process. Such drivers are not able to perform packet processing at expected high speed, especially when 10/40/100G Ethernet generation cards are used on a Linux System.

A Poll Mode Driver consists of APIs, running in user space, to configure the devices and their respective queues. In addition, a PMD accesses the RX and TX descriptors directly without any interrupts (with the exception of Link Status Change interrupts) to quickly receive, process and deliver packets in the user’s application.

Poll Mode drivers are involved in NIC configuration. They are exposing NIC configuration registers into host memory area which is directly reachable from user space.

In short, Poll Mode Drivers are user space pthreads which:

Hence user applications can configure directly the NIC cards they are using from Linux user space where they are running.

A first configuration phase is using Poll Mode Drivers and DPDK library to configure DPDK rings buffers into Linux user space. Next, incoming packets will be automatically transferred with DMA (Direct Memory Access) mechanism from NIC physical RX queues in NIC memory to DPDK RX rings buffer in host memory. DMA (Direct Memory Access) is also used to transfer outgoing packets from DPDK TX rings buffer in host memory to NIC physical TX queues in NIC memory. DMA offloads expensive memory operations, such as large copies or scatter-gather operations, from the CPU.

Direct Memory Access (DMA) allows PCI devices to read (write) data from (to) memory without CPU intervention. This is a fundamental requirement for high performance devices.

DMA is a mechanism that is using a specific hardware controller to manage read and write operations into the main system memory (RAM: Random Access Memory). This mechanism is totally independent of the central processing unit (CPU) and does not consume any CPU resource. A DMA transfer is used to manage data transfer. DMA transfer is triggered by the CPU and is working in background using the specific hardware resource (DMA controller).

DPDK rings and NIC buffers are synchronized with DMA. Thanks to this synchronization mechanism, DPDK application can access transparently to NIC packets in user space reading or writing data in DPDK rings.

Input–Output Memory Management Unit (IOMMU) is a memory management unit (MMU) that connects a Direct Memory Access (DMA) capable I/O bus to the main memory.

In Virtualization, an IOMMU is re-mapping the addresses accessed by the hardware into a similar translation table that is used to map guest virtual machine address memory to host-physical addresses memory.

IOMMU provides a short path for device to get access only to a well scoped physical device memory area which corresponds to a given guest virtual machine memory. IOMMU helps to prevent DMA attacks that could be originated by malicious devices. IOMMU provides DMA and interrupt remapping facilities to ensure I/O devices behave within the boundaries they’ve been allotted.

Intel has published a specification for IOMMU technology as Virtualization Technology for Directed I/O, abbreviated as VT-d.

In order to get IOMMU enabled:

GRUB configuration example with IOMMU Passthrough enabled:

DPDK Library includes Poll Mode Drivers (PMDs) for physical and emulated Ethernet controllers which are designed to work without asynchronous, interrupt-based signaling mechanisms.

Lots of other NIC are supported by DPDK (cf http://doc.dpdk.org/guides/nics/overview.html).

Different PMDs may require different kernel drivers in order to work properly (cf Linux User space device enablers). Depending on the PMD being used, a corresponding kernel driver should be loaded and bound to the network ports.

This is also preferable that each NIC has been flashed with the latest version of NVM/firmware.

Linux kernel version 2.6 introduced the User Space IO (UIO) loadable module. UIO is a kernel-bypass mechanism which provides an API that enables user space handling of legacy interrupts (INTx).

UIO has some limitations:

UIO only supports legacy interrupts so it is not usable with SR-IOV and virtual hosts which require MSI/MSI-X interrupts.

Despite these limitations, UIO is well suited for use in virtual machines, where direct IOMMU access is not available. In such a situation, a guest instance user space process is not isolated from other processes in the same instance. But the hypervisor can isolate any guest instance from others or hypervisor host processes using IOMMU.

Currently, two UIO modules are supported by DPDK:

DPDK specific UIO Kernel module is loaded with insmod command after UIO module has been loaded:

While a single command is needed to load Linux Generic UIO module:

DPDK specific UIO module could be preferred in some situation to Linux Generic UIO module (cf: https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html)

Virtual Function I/O (VFIO) kernel infrastructure was introduced in Linux version 3.6.

VFIO provides a user space driver development framework allowing user space applications to interact directly with hardware devices by mapping the I/O space directly to the application’s memory.

VFIO is a framework for building user space drivers that provides:

VFIO exposes APIs which allow to:

VFIO driver is an IOMMU/device agnostic framework for exposing direct device access to user space, in a secure, IOMMU protected environment. For bare-metal environments, VFIO is the preferred framework for Linux kernel-bypass. It operates with the Linux kernel’s IO.

MMU subsystem is used to place devices into IOMMU groups. User space processes can open these IOMMU groups and register memory with the IOMMU for DMA access using VFIO ioctl calls. VFIO also provides the ability to allocate and manage message-signaled interrupt vectors.

A single command is needed to load VFIO module:

Despite VFIO has been created to work with IOMMU, VFIO can be also be used without (this is just as unsafe as using UIO).

VFIO is generally the preferred Linux user space device enabler to be used because it supports IOMMU to protect host memory. When a real hardware PCI device is attached to host system and IOMMU is used with VFIO, all the reads/writes of that device done in user space by the DPDK application will be protected by the host IOMMU.

But there some is few exceptions. Below is Intel recommendation for the choice of the Kernel driver to be used with DPDK:

https://software.intel.com/content/www/us/en/develop/articles/memory-in-dpdk-part-2-deep-dive-into-iova.html

NUMA means Non-Uniform Memory Access systems

A traditional server has a single CPU, a single RAM and a single RAM controller.

A RAM can be made of several DIMM banks in several sockets, all being associated to the CPU. When the CPU needs access to data in RAM, it requests it to its RAM controller.

Recent servers can have multiple CPUs, each one having its own RAM and its own RAM controller. Such systems are called NUMA systems, or Non-Uniform Memory Access. For example, in a server with 2 CPUs, each one can be a separate NUMA: NUMA0 and NUMA1.

NUMA nodes architecture.

When CPU0 needs to access data located in RAM0, it will go through its local RAM controller 0. Same thing happens for CPU1.

When CPU0 needs to access data located in the other RAM1, the first (local) controller 0 has to go through the second (or remote) RAM controller 1 which will access the (remote) data in RAM 1. Data will use an internal connection between the 2 CPUs called QPI, or Quick Path Interconnect, which is typically of a high enough capacity to avoid being a bottleneck, typically 1 or 2 times 25GBps (400 Gbps). For example, the Intel Xeon E5 has 2 CPUs with 2 QPI links between them; Intel Xeon E7 has 4 CPUs, with a single QPI between pairs of CPUs.

The fastest RAM that the CPU has access to is the register, which is inside the CPU and reserved to it.

Beyond the register, the CPU has access to cached memory, which is a special memory based on higher performance hardware.

Cached memories are shared between the cores of a single CPU. Typical characteristics of memory cache are:

If the CPU needs to access data that is in the main RAM, it has to use its RAM controller.

Access to RAM takes typically 170 CPU cycles (the green line in the diagram). Access to the remote RAM through the remote RAM controller typically adds 200 cycles (the red line in the diagram), meaning RAM latency is roughly doubled.

When data needed by the CPU is located both in the local and in the remote RAM with no particular structure, latency to access data can be unpredictable and unstable.

A single physical CPU core with hyper-threading appears as two logical CPUs to an operating system.

While the operating system sees two CPUs for each core, the actual CPU hardware only has a single set of execution resources for each core.

Hyper-threading allows the two logical CPU cores to share physical execution resources.

The sharing of resources allows two logical processors to work with each other more efficiently and allows a logical processor to borrow resources from a stalled logical core (assuming both logical cores are associated with the same physical core). Hyper-threading can help speed processing up, but it’s nowhere near as good as having actual additional cores.

Memory is managed in blocks known as pages. On most systems, a page is 4KB. 1MB of memory is equal to 256 pages; 1GB of memory is 256,000 pages, etc. CPUs have a built-in memory management unit that manages a list of these pages in hardware.

The Translation Lookaside Buffer (TLB) is a small hardware cache of virtual-to-physical page mappings. If the virtual address passed in a hardware instruction can be found in the TLB, the mapping can be determined quickly. If not, a TLB miss occurs, and the system falls back to slower, software-based address translation. This results in performance issues. Since the size of the TLB is fixed, the only way to reduce the chance of a TLB miss is to increase the page size.

Virtual memory address lookup slows down when the number of entries increases.

A huge page is a memory page that is larger than 4Ki. In x86_64 architecture, in addition to standard 4KB memory page size, two larger page sizes are available: 2MB and 1GB.

Contrail DPDK vrouter can use both or only one huge page size.

An Operating System is using a scheduler to place each single process and/or threads it has to run onto one CPUs offered by a host.

There are two kinds of scheduling, cooperative and preemptive. By default, Linux scheduler is using a cooperative mode.

In order to get a CPU booked for a subset of tasks, we have to inform the Operating System scheduler not to use these CPUs for all the tasks it has to run.

These CPUs are told: "isolated" because they are no more used by the OS to process all tasks. In order to get a CPU isolated several mechanisms can be used:

Isolation and pinning are two complementary mechanisms that are proposed by Linux OS:

Tasks to be run by an operating system must be spread across available CPUs. These tasks in a multi-threading environment are often made of several processes which are also made of several threads.

A thread’s CPU affinity mask determines the set of CPUs on which it is eligible to run.

Linux systemd is a software suite that provides an array of system components for Linux operating systems. Its primary component is an init system used to bootstrap user space and manage user processes.

CPUAffinity parameter restricts all processes spawned by systemd to the list of cores defined by the affinity mask.

taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity.

We can retrieve the CPU affinity of an existing task:

Or set it:

Virtual IOMMU (vIOMMU) is allowing to emulate IOMMU for guest VMs.

vIOMMU has the following characteristics:

In order to get a virtual device working with a virtual IOMMU we have to:

Following mechanisms can be used to create vIOMMU memory mappings:

The integration between the virtual IOMMU and any user space network application like DPDK is usually done through the VFIO driver. This driver will perform device isolation and automatically add the memory (IOVA -to GPA) mappings to the virtual IOMMU.

The use of hugepages memory in DPDK contributes to optimize TLB lookups, since a fewer number of memory pages can cover the same amount of memory. Consequently, the number of Device TLB synchronization messages drop dramatically. Hence, the performance penalty TLB lookups is lowered.

Cf: https://www.redhat.com/en/blog/journey-vhost-users-realm

https://wiki.qemu.org/Features/VT-d

Virtio-pmd driver, is a DPDK driver, built on the Poll Mode Driver abstraction, that implements the virtio protocol.

Vhost user protocol moves the virtio ring from kernel all the way to userspace. The ring is shared between the guest and DPDK application. QEMU sets up this ring as a control plane using unix sockets.

If the both the host server guest virtual machine are DPDK there are no VMExits in the host for guest packets processing. Guest virtual machine uses virtio-net PMD driver and performs packets polling. So. There is nothing running in kernel here, so there are no system calls. Since both system calls and VM Exits are avoided, the performance boosts significantly. It will be an order higher.

When a DPDK application is running into a guest Virtual Machine, a mechanism has to be used to expose one of the host physical NIC to this guest in order it gets access to the physical network.

IOMMU protects host memory against malicious or bug writes which can corrupt host memory at any time. But, when a physical device is assigned to a guest virtual machine without vIOMMU usage, the guest memory address space is totally exposed to the hardware PCI device.

A PCI device can be assigned to a guest in order to be used by a guest DPDK application. By leveraging VFIO driver in the host kernel we provide a direct access to an assigned physical NIC from this guest protected with IOMMU.

Next, by leveraging VFIO driver in the guest kernel we provide a direct access to the assigned physical from this guest user space. vIOMMU is providing a secure mechanism to manage DMA transfer between an assigned physical hardware and hosted guest virtual instance memory area.

This use case is almost the same as PCI passthrough. VFIO and IOMMU are used to expose a SRIOV virtual function directly to a guest VM.

An additional Physical function driver which is vendor specific is used to manage the virtual function creation on the physical NIC. This driver is used by a Virtual Machine Manager (like libvirt) to create the virtual function before the virtual instance is spawned.

Physical incoming packets are directly copied in guest memory without involving the host server. SR-IOV only allow to share a physical NIC between several guests but does not change the packet processing path provided by PCI passthrough.

With virtio full hardware offloading, both the virtio data plane and virtio control plane are offloaded to the NIC hardware. The physical NIC must support:

Hence once the guest memory is mapped with the NIC using virtio physical device passthrough, the guest communicates directly with the NIC via PCI without involving any specific drivers in the host kernel.

Guest VM packet processing is directly performed in NIC hardware like but presented to the guest instance like a regular virtio emulated interface. Guest VM does not make any difference between a virtio emulated interface and an assigned physical virtio NIC, as they are exposed with the same virtio driver frontend in the guest.

virtio device passthrough

Virtio device passthrough can be implemented onto a NIC which is supporting or not SR-IOV.

Like other physical device assignment technics presented in this book, VFIO and IOMMU are used to present the physical device NIC into the guest VM user space.

Hence, such a virtio physical NIC can be used by a DPDK application running into a virtual instance. But, like other virtio device passthrough has also the same limitations for SDN. As the host operating system is totally by passed by this mechanism, we cannot interconnect instances using such NIC interface with a SDN virtual router or switch.

The main advantage of Virtio device passthrough is the flexibility it provides for a virtual instance to use transparently either a real physical interface or an emulated one. It offers an Open public specification, which provide device fully independent of any specific vendor.

Virtio full HW offloading, can support live migration thanks to virtio, which is not possible to achieve without any specific implementation with SR-IOV.

But in order to be able to support such a feature, latest virtio specifications (1.1 version) must be implemented onto both QEMU and the NIC hardware used on the cloud infrastructure.

Like full hardware offloading, virtual Data Dath Acceleration (vDPA) aims to:

vDPA is presenting a generic control plane through a software piece which provides an abstraction layer on top of physical NIC.

Like Virtio full hardware offloading, vDPA build a direct data path between the gest network interface and the physical NIC, using the virtio ring layout. But for the control path a generic vDPA driver (mediation driver) is used to translate the vendor NIC driver/control-plane to the VirtIO control plane, in order to allow each NIC vendor to keep using its own driver.

It allows NIC vendors to support virtio ring layout at smaller effort keeping wire speed performance on the data plane.

virtio datapath acceleration

vDPA is requiring a vendor specific "mediation device driver" to be loaded in the host operating system.

A NIC card generation commonly named "smart NIC" are highly customizable thanks to the last evolution provided by some new capabilities (FPGA, P4).

It makes possible to envisage SDN vSwitch/vRouter dataplane function to be moved into the NIC card keeping only the controle plane function into the host operating system.

For Contrail solution, this is made by offloading several Contrail vRouter tables including:

It allows to accelerate lookups and forwarding actions that are directly performed into the NIC.

SDN packet processing is fully done into the NIC card, no more host CPU processing is involved in packet processing.

Two implementations are proposed by Metronome:

SRIOV + SmartNIC:

vDPA + Smart NIC:

Berkeley Packet Filter (BPF) was designed for capturing and filtering network packets that matched specific rules. In last years extended BPF (eBPF) has been designed to take advantage of new hardware (64 bits usage for intance). An eBPF program is "attached" to a designated code path in the kernel.

eXpress Data Path (XDP), uses eBPF to achieve high-performance packet processing by running eBPF programs at the lowest level of the network stack, immediately after a packet is received. XDP.

XDP support is made available in the Linux Kernel since version 4.8, while eBPF is supported in the Linux Kernel since version 3.18.

XDP requires:

XDP packet processor performs:

BFP programs:

BFP program main actions are :

XDP is also able to offload an eBPF program to a NIC card which supports it, reducing the CPU load.

XDP and eBPF does not require:

PS: eBPF rules are also supported in DPDK application.

https://www.redhat.com/en/blog/using-express-data-path-xdp-red-hat-enterprise-linux-8

The vRouter packet processing pipeline is described in the picture below.

There are various tables and engines in action in this pipeline. Some of the important tables in this pipeline are flow table, route table, NH table and the MPLS/VXLAN table. The vRouter agent programs these tables based on the forwarding state it receives from the control node and also based on its own internal processing. Each packet, depending on which interface it is coming from, is subjected to the desired processing.

At a high level, all packets enter from an interface called ‘vif’. The vifs are nothing but one of the vRouter interfaces that we described previously. Example: tap interface, physical interface, vhost0 interface, agent interface etc. Depending upon the configuration of that interface, it enters different pipeline stages, doing lookups in different tables and based on what actions are defined in each stage, the packets are modified accordingly.

At the end of the processing, it is sent to another vRouter interface or vif after encapsulation or decapsulation. This is a fairly generic pipeline and the agent stitches this based on the rich feature set that the contrail cluster is configured.

Another important aspect of vRouter is that of forwarding modes. The vRouter can work in two modes - flow mode (bottom pipeline on the diagram above) or packet mode (top on the diagram above). By default, Contrail works in flow mode. This means that, vRouter keep track of every single flow traversing it. Depending on the flow action, it can either forward the packet or drop it. In the packet mode, vRouter bypasses the flow table and directly uses the nexthop to treatment that needs to be given to the packet. Example: If the nexthop is tunnel next hop, the packet is encapsulated in a tunnel header and forwarded into an outgoing interface.

Contrail supports three kinds of vRouter deployments:

DPDK vRouter is a userspace application as mentioned previously. It is comprised of multiple pthreads, which are also called lcores (logical cores) in DPDK terminology. Each pthread has a specific role to perform. The lcores run in a tight loop, also called the poll mode. They can exchange packets among themselves using DPDK queues. Each lcore has a receive queue which can be used by other lcores to enqueue packets which needs to be processed by that lcore. They also poll different vRouter interfaces queues like - physical, VM and tap.

vRouter is a multi-threaded user-space application. It spawns several pthread or lcores which run in a tight ‘while’ loop. Each lcore is responsible for a specific task. The different types of lcores are:

Forwarding lcores are responsible for polling the physical and virtual interfaces. Physical interfaces can be a bonded interface also. In addition, they can do the vrouter packet processing which is briefly illustrated in the section “vRouter packet processing Pipeline”. These lcores can assume the role of both polling and processing.

These lcores are spawned by the vRouter with a well-defined CPU list. It gets the CPU list as a “core mask” using the “taskset” linux command.

Example: taskset 0x1e0 /usr/bin/contrail-vrouter-dpdk --no-daemon

The hex representation of 0x1e0 is as follows –

This will make the vRouter spawn 4 forwarding cores and they will be pinned to CPUs 1,2,3,4

The first forwarding is named lcore10, the next one is named lcore11, and so on. Hence if a DPDK vRouter has been configured with 4 polling and processing CPU into its CPU list, 4 threads will be launched: lcore10, lcore11, lcore12 and lcore13.

The below is the output which lists the threads running in vRouter, it’s names and also it’s PIDs

Service lcores are responsible for handling all vRouter interfaces other than workload (VM) interfaces and physical interfaces. It also hands other book-keeping and miscellaneous tasks for vRouter like timer management and vhost-user control path. By default, they are not pinned to any physical CPU.

Most of the service lcores make use of user sockets to talk to other processes in the system like agent, qemu (VM) and linux stack.

During initialization of the NIC interface, vRouter configures it with the same number of queues as the number of forwarding cores it has. For example, if the vRouter has 5 forwarding cores, the number of Rx queues it configures to the NIC is 5.

A vif queue is made up of two DPDK rings:

Packets stored in vif RX rings are polled by a forwarding lcore. There is a one-to-one mapping between forwarding cores and the NIC’s Rx queues. Then the polled packets are processed by a the same lcore or different one and then pushed to a target vif’s TX ring.

Each lcore10 and higher started by a DPDK vRouter is a polling and a processing thread. They are running onto a single CPU listed defined by CPU_LIST variable.

Incoming overlay encapsulated packets are received on the Compute physical Network Interface Card, usually a Bond made up of 2 NICs, used for user packets transport.

Incoming Overlay packets are placed into Physical NIC queues using DPDK RSS (Received Side Scaling) hashing algorithm. A vRouter startup are created (with the help of the physical NIC PMD) as many DPDK queues (both RX and TX rings) as the number of vRouter allocated polling and processing cores.

The RSS hashing algorithm for MPLSoGRE only use 3 tuple values: IP source, IP destination, protocol number. Unfortunately, the entropy of these 3 values is low when GRE is used.

Indeed, the 3 tuple values is kept the same between 2 different compute nodes.

All packets coming from different Virtual Machines located on a same compute node will be bound to the same 3 tuple value. Hence, the hashing algorithm will provide an identical value for all network flows coming from each single compute.

Consequently, all packets coming from Virtual Machines located on a same compute will be received only in one DPDK RX ring of the vif0/0 interface (vRouter interface connected to the underlay network).

So, incoming MPLS GRE overlay packets are not well balanced onto the different polling and processing threads (lcores) the vRouter is fitted with. Therefore, when MPLS GRE overlay is used, it has been chosen to perform the packet processing (packet transformation and delivery in a vif TX ring) onto a different lcore than those used for the packet polling (retrieve a packet from a vif0/0 RX ring).

A DPDK pipeline model is then used. A first lcore is only performing packet polling, a second one is performing the packet processing. Some internal queues are setup in order to store packets that have been polled by the polling lcore thread and that are waiting to be processed by the processing lcore thread.

A hash algorithm is applied onto the decapsulated packet (inner packet) in order to select one of the internal queues that are each handled by a single processing lcore thread.

Thanks to this mechanism, even if few compute nodes are used into the physical infrastructure and user packets carried with MPLS over GRE overlay protocol, all vRouter allocated CPU are used.

When an UDP overlay protocol is used (MPLS over UDP or VxLAN) we have a better entropy, 5 tuples from values: IP source, IP destination, source port and destination port, protocol. Indeed, even if few computes are used, the sending compute can create diversity using some distinct values in the UDP source port of overlay packet.

Different network flows coming from a same virtual remote machine will generate different RSS hash results.

Consequently, incoming overlay packets are balanced onto all the DPDK RX rings configured for the physical interface. It is useless to split polling and processing steps. Therefore, when an UDP overly protocol is used to transport user packets between compute nodes, the vRouter is using the same lcore for both polling and processing steps of each packet.

It is more efficient to use UDP overlay protocols. Performance reached with a same DPDK vRouter configuration is higher when an UDP overlay protocol is chosen instead of MPLS over GRE.

NIC connected to vRouter can be configured to several queues.

Each NIC queue is automatically pinned to a single vRouter polling and processing thread (lcore10 and higher). Consequently, when a NIC is configured with only a single Q, all incoming and outgoing packets are processed by a single vRouter polling and processing threads.

In order to avoid binding all single queue interfaces on the same polling and processing thread; each interface queue is pinned to a distinct vRouter lcore into a round robin manner when each interface is created. Single queue vif0/3 is automatically pinned to lcore 10, single queue vif0/4 is automatically pinned to lcore 11, and so on.

Hence the vRouter whole CPU power is automatically distributed among all the single queue interfaces. This distribution is automatically defined for each interface and is kept unchanged during all the interface life duration.

When a NIC is configured with several queues, each single queue is bound to a distinct polling and processing thread (lcore). Hence the vRouter whole CPU power is automatically distributed among all the queues on each multi queue interface.

Even if there is no hard rule that prevent a user to configure a different number of queue on a NIC as the number of lcores (polling and processing threads) configured on the vRouter; the best scenario is to configure each multi queue NIC with the same number of queue as the number of configured polling and processing threads on the vRouter.

In this book, we’ve been focusing on DPDK vRouter that runs in each individual compute node, which basically runs in a relatively standalone mode. but if you look at forwarding plane as a whole, they are actually a distributed system. In fact as we’ve briefed in chapter 1, the whole TF cluster is a complex distributed system involving a lot more different software modules especially in control plane. Again, each of the software module can be a completely different distributed system by themselves. The cassandra database that TF cluster uses is one such example. Explaining and understanding details about how things works in distributed system is never easy, and so is the installation process. It won’t be a surprise if you run into some installation issues in your lab. Generally speaking, it is always much more efficient to follow a detailed, verfied process with step by step instructions to "avoid" the issues, than starting with a "try-and-see" mode and then try to fix the issues.

Currently, TF cluster has been widely integrated with many majority deployment systems and platforms. Therefore depending on your envrionment there can be totally different ways of installing contrail system. Here is a incomplete list of currently supported installation methods:

For example, the second method, you can install contrail with Redhat openstack platform director 13 (RHOSPd), which is a toolset based on the OpenStack project TripleO (OOO, OpenStack on OpenStack). An TF environment built out of RHOSPd uses concepts of undercloud and overcloud. Basically undercloud is a single server containing complete OpenStack components, whose role is just to deploy and manage an overcloud, which is a tenant-facing environment that hosts the "resulting" openstack and TF nodes. This deployment is currently used by quite a few major service providers in production.

However, the installation process of such a deployment involves the understanding of RHOSPd, TripleO, and of lots of different types of networks isolation topologies, which add many uncessary complexities to our lab setup. In this section, we’ll give a detail steps about the first method - installing contrail with openstack and kolla ansible.

Kolla is an OpenStack project which provides tools to build container images for OpenStack services. Kolla Ansible provides Ansible playbooks to deploy the Kolla images. The contrail-kolla-ansible playbook works in conjunction with contrail-ansible-deployer to install OpenStack and Contrail Networking containers.

TODO: some text here

To enable bond interface in centos, under /etc/sysconfig/network-scripts/ of all nodes where bond interface is needed, add these configuration files:

Then restart network service to invoke these configurations:

Once the restart is successful, you should see bond0 interface appearing in all nodes with one of these IP addresses in each node: 8.0.0.1~4. Now we should have the IP connectivities in both management network and fabric network.

Next we’ll need to install ansible and use it to automate the rest part of the installations. The most part of ansible’s magic is done throught its playbooks, and configuration for all plays is done in a single file with a default name instances.yaml. This configuration file has multiple main sections. We’ll go over some of the main parameters in this file and then introduce the steps to run the playbooks.

Once the yaml file is carefully prepared, the installation process is relatively easy. Basically we just need to install some pre-requisite software packages, such as python libraries, git and ansible tools. git is required to clone a github repository which includes all ansible playbooks. Then, we use ansible to automate the installation in all nodes based on the playbooks and our configuration file instances.yaml. The detail steps are here:

Once everything succeeds, you will have an up and running 4 nodes contrail cluster (1 controller node and 3 vRouter/compute node). You can login to the setup through webUI or ssh session to check the system running status.

Here is the contrail web UI for a working setup:

You can also login to each individual nodes with ssh, and run contrail-status command to verify the running status of each components of it.

If everything works, congratulations! You now have your own lab to play with. Next, we’ll go over the steps of setting up testing tools to send and receive traffic - the PROX and rapid script.

PROX (Packet pROcessing eXecution Engine) is an OPNFV project application built on top of DPDK. It is capable of performing various operations on packets in a highly configurable manner. It also support performance statistics that can be used for performance investigations. Because of the rich feature set it supports, it can be used to create flexible software architectures through small and readable configuration files. In this chapter we’ll introduce how to use it to test vrouter performance in DPDK environment.

In a typical test you need two VMs running PROX. VM1 is generating packets, sending them to VM2 which will perform a "swap" operation on all packets, so that they are sent back to VM1.

In this book we will call them "gen" and "swap" VM respectively. One special feature we used here is that, the "swap" PROX is configured in such a way that, once receives the packets sent from the generator, it will "swap", or "loop" them back to the generator VM, so the latter can collect them and calculate how much traffic got forwarded by the DUT - in our case it is the DPDK vRouter.

Rapid(Rapid Automated Performance Indication for Dataplane) is a groups of "wrapper" scripts interacting with PROX to simplify and automate the configuration of PROX. It is a set of files and scripts offering an even easier way to do a sanity check of the dataplane performance.

rapid is very powerful and configurable. A typical workflow is like below:

The rapid scripts are typically installed in a third VM, called "jump" VM in this book. The purpose of this VM is to control the traffic generator to start, stop, pause the test as well as collecting the statistics.

A typical prox and rapid testing setup looks like this:

The test setup consists of three compute nodes, running the above mentioned 3 VMs respectively:

Here is a brief summary of hardware requirements for different VM:

As mentioned earlier, to perform the test we need two VM both running PROX. One sending traffic and the other one receive and swap it back. Same exact PROX application is running but with different configuration files.

Apparently, the IP level connectivity is required in order for the two VM to be able to exchange packets with each other. In our case, the two VM will be spawned by openstack nova. Needless to say, all supporting objects and resources associcated to the VMs, like IPAM, subnet, virtual-network and VM flavor (size of CPU/memory/storage/etc), also need to be created out of openstack infrastructure, either from horizon webUI or openstack CLIes. A quick list of the common tasks are listed here:

On top of these, installing PROX inside of the VMs, like with many other open source projects, often requires downloading the source code and compile it in your platform. That means you download the PROX source codes, compile it to get the execute, then configure and run the application. In this section we’ll introduce how PROX is installed in our setup we built for this book, You can find more details in PROX website here: https://wiki.opnfv.org/display/SAM/PROX+installation

The software and CPU model we use here are shown below:

In our lab setup the VM OS is the same as the host, and the emulated CPU Model is Intel Xeon E3-12xx:

PROX is a dpdk application. When running, it connects to the DPDK libraries to implement most of its features. Therefore to build it we need a DPDK environment.

The steps to build DPDK in our setup is as below:

Add the following to the end of ~/.bashrc file

Now with DPDK libraries built, we can start to download, extract and build the PROX application. Here are the steps:

When make succeeds, the compiled binary PROX will be available in build folder of current directory.

We’ll demonstrate this later.

The set of sample configuration files can be found in: ./config folder. Sample configs of PROX functioning as the "generator" is available in ./gen/ folder.

Assuming the current directory is where you’ve just built PROX, we can just launch PROX with a proper configuration file.

When it runs, a ncurse based UI will pop up and through it you will see update about the running states in real time. We’ll give an example on this later.

Rapid scripts can be downloaded from here: https://github.com/opnfv/samplevnf/tree/master/VNFs/DPPD-PROX/helper-scripts/rapid The scripts were developed in python, so you can run them directly and no need to compile.

We have just introduced the steps of manually compiling PROX from source code. We also has assumed you know how to perform a list of tasks to create all necessary objects required by the VMs from openstack. Doing this one time is not a big deal. Suppose you are working in a dynamic environment where you often need to:

Repeating these manual steps will become a tedious and even painful job. You will soon prefer to be able to simplify the building, creation and configuration of PROX, as well as creating all necessary openstack resources. In openstack environment the NO. 1 choice for automation is heat. With heat, typically all tasks are programmed in a template file, with calls all parameters from another environment file. In appendix, we provide all sample template file as long as environment file and associcated scripts, which are tested and proved to be working fine in our setup. You can use them as a starting point, then make necessary customizations based on your environment to build your owen automation. The virtual machine, where the tools are running, including rapid scripts and PROX DPDK application that is pre-compiled in it, has also been built as an image . With all these automations carefully designed and tested, all what we need to do now becomes much simpler:

If everything goes well, you will have your whole PROX testing environment available in just a few minutes. The detail steps are listed below:

Wait for a few minutes and use openstack stack list command to check the stack creation status.

Once succeeded, you can use different sub-command of openstack stack command to retrieve the parameters of the stack components.

The image has been configured with a root password Login c0ntrail123. So all 3 VMs, once up and running, will inheritage the same login credential. In contrail/openstack integration environment There are a few common ways to access a VM running in a specific compute node:

In our test we didn’t configure any floating IP, so we will use console and meta_ip_address to access the VM. To access VM console use virsh console command from nova_libvirt docker in the compute node:

Comparing with console, ssh session is usually preferred. Let’s take a look at each VM’s allocated interface IPs with openstack server list command:

let’s take our "jump" VM stack2-jump for instance. Openstack allocated an IP address 192.168.0.106 to it’s tap interface from the stack2-control virtual-network. However, this IP address is not directly reachable from the host. In order to ssh into the VM, we need to first locate the meta_ip_address allocated to the VM’s tap interface, or more specifically, the vif interface in vRouter. We can use vRouter vif command to confirm which vif interface has this IP.

Good. vif0/3 has the IP, so this vif connects to the tap interface of our jump VM. In contrail vRouter, for each vif there is also a "hidden" meta_data_ip of "169.254.0.N", wherre N is the same number as in the vif0/N. Therefore in this case our meta_data_ip is "169.254.0.3". Let’s try to start a ssh session into it:

It works. The benefit of this approach is that, not only the interaction with the VM is much faster, but also it supports file copies with scp tool. Remember in many cases the VM does not has any Internet connection, so in case you need to copy files into (or out of) the VM, the meta_data_ip method will be especially useful.

With the stack created and all VMs up and running, we now can introduce how to run test with rapid. Remember rapid is installed in the "jump" VM, so we’ll need to execute the script from there.

On Jump VM, go to /root/prox/helper-scripts/rapid/ folder, where you can locate a python script named "runrapid.py". To run test you can just run it without any other parameters:

This will start rapid script and send traffic for 10 seconds by default. the period of time for sending traffic can be adjusted by --runtime option:

A few other command line options are supported, which can be listed by -h:

A typical runrapid.py script execution looks like this:

You can see that some preparation work were done before the actual test are started: . First, the script read 3 files, rapid.env, basicrapid.test and machine.map. The env file provides IP/MAC information of the gen and swap VM, and the .test file defines all detail behavior of the test.

The script is highly configurable. In appendix We provide a sample "basicrapid.test" that we use in our lab. You can start with it and fine tune based on your need. For example, in section [test2] of the file you can change number of flow and packet size to define different test scenarios.

OK. We just introduced rapid. The script support very extensive options in the configuration files which beyond the scope of this book, but we’ve got the idea how it works basically. Please remember that rapid and PROX and two different applications. Rapid script does all the magics and make your life easier through automation of PROX, and PROX is the foundation application that does the "real" works. In fact, PROX can run tests just fine without Rapid. To launch PROX and start traffic, in the "gen" VM’s home folder (root in our case) start this command:

PROX will parse its configuration file /root/gen.cfg and start to boot. from the booting messages in the screen we can learn its booting sequences:

You will end up with a ncurse based UI like below:

The display shows per task statistics which include: estimated idleness, per second statistics for packets received, transmitted or dropped; per core cache occupancy, cycles per packet, etc. These statistics can help pinpoint bottlenecks in the system. This information can then be used to optimize the configuration. There are quite a few other features include debugging support, scripting, Open vSwitch support, etc. Refer to PROX website for more details. For now, let’s look at how the traffic flows. Right now from the screenshot above we only see traffic being sent, but nothing gets received yet. Reason is we are now running PROX manually and we only starting the "gen" side, which is the traffic "sender" only. We need to start the "swap" VM as well as a "receiver", who will also "loop" the traffic back to the sender, so our first PROX application will see some "RX" statistics. Let’s do that. On the compute where "swap" VM is installed, execute the same prox command line, except this time we pass a different configuration file named swap.cfg:

Here you will end up with a similiar ncurse based text UI, after similiar booting process as of the sender. Once our "swap" end of PROX is up and running, immediately you will see both "RX" and "TX" counters keep updating on both side of the traffic:

That concludes our discussion of PROX and rapid as our testing tools. We’ll use this tools intensively in the rest of this chapter to generate different kinds of traffic in each tests. With the traffic running, we can dig deeper to understand the rules we’ve introduced about how vRouter works. Next we’ll introduce some of the commonly used tools that are designed for, or especially useful for verfications in DPDK vRouter environment.

"contrail-tools" is a docker container located in the compute node, where all of the vRouter tools and utilities are available. Apparently, from the user perspective, this is more convenient than distributing tools into multiple containers. This design was introduced a few releases before contrail networking R2008. As more and more existing tools migrated into it and new tools added in, this container now really becomes a centralized "tool box", which you’d like to open whenever you want to check any running states of the vRouter dataplane. Let’s first take a look at how to "open" this "box".

To enter the container, just run contrail-tools script (same name as of the docker) in a compute node.

Now you are inside of the container. From here you can test all of the old vRouter tools you may have been familiar with, for example, to print the packet dropping statistics:

When you are done, just exit the docker and it will be killed.

you can also pass the tool command as parameters to the script, execute the command, get its output, exit the docker, all with one go.

As the time of the writing of this book, there are nearly 20 tools available in this container. Let’s take a look at what’s in the package.

First, in the container we’ll locate the package name:

Then, based on the package name, we can list all available tools in it:

In previous chapters you’ve read about dpdk_nic_bind.py script, which is a tool to tell bind a specific driver for a NIC. In the rest of this section, we’ll introduce some more tools that is especially useful in DPDK environment.

The first one from our contrail DPDK "tool box" is vif command. Before talking about it, let’s see how do we list all interfaces in the compute running DPDK vRouter. Let’s first try the linux ip or ifconfig command in our DPDK compute running PROX gen VM:

Well, we do see some interfaces got printed:

However, some most important interfaces are not shown at all:

If we compare with what we would see with the same ip command in a kernel mode vRouter compute without DPDK, we will see the big differences:

Here except lo, management interface and whatever we saw from the DPDK compute, we also see these all other important interfaces:

The reason we see these differences, as we’ve mentioned many times throughout this book, is that when DPDK is in charge of the NIC card, linux kernel is mostly "bypassed". The NIC card’s feature and functions are exposed by another special driver directly to the user space PMD driver running in DPDK layer, so the traditional applications, whichever relies on the interfaces sitting in linux kernel to do its job, are no more useful.

We’ll talk more about this later. for now, let’s look at the vif command with -l|--list and --get option. vif --list lists all interfaces located in the vRouter and --get just retrieves one of them. Here is the capture from the same DPDK compute:

Here the vRouter interfaces are:

Now you should understand the importance of vif command, especially in DPDK vRouter. It shows interfaces from vRouter’s perspective, and reveals the one to one connection mapping between vRouter and fabric or VM tap interface. The latter would be "invisible" otherwise.

Besides that, it also prints other important information. The Vrf numbers and packet counters are the most commonly used data points. Among various counters, usually we focus on the RX/TX packets/bytes counters which displays data received or sent in packets or bytes. Depending on your environment, sometime you may also see non-zero numbers in RX/TX queue packets/errors counter that gives inter lcore packet statistics. It is usually happens when two lcores are involved in the packet forwarding path. We’ll use this command intensively in the rest of this chapter, and we’ll analyze these counters and use them to understand some of the important vRouter working mechanisms.

vif tool also support some other options, use --help to print a brief list of all currently supported options.

We won’t talk about each every options and their usage and usually you don’t need to know anything except --get and -l|--list. There is one more (--add) which we’ll talk about shortly. For others you can refer to https://www.juniper.net/documentation/en_US/contrail20/topics/task/configuration/vrouter-cli-utilities-vnc.html for more details.

Next, let’s look at two useful scripts that are developed based on vif command: dpdkvifstats.py and vifdump.

We’ve talked about vif and dpdkvifstats.py tools. Now let’s introduce a relatively new tool that can be used to investigate lower level details of DPDK interfaces. dpdkinfo is introduced since Contrail 20.08. Using this tool Contrail operators can collect more information about DPDK vRouter fabric interface internal status, connectivity (physical NIC bond), DPDK library information, and some other statistics.

Let’s first run the tool with -h to get a brief menu of it:

From this help information we can see it provides information about DPDK interface in multiple areas. In this rest of this section, let’s take a look at some of the most useful options, they are:

There are some other options like --app|-a, --mempool|-m we won’t introduced in this book, and the list of supported functions may grow in each future releases. But you will get the basic idea of its usage and you can refer the official documents for other usage informations.

TODO: this is copied and rewritten based on Laurent’s chapter 5. will need to capture in same environment and rewrite again.

Contrails DPDK vrouter dataplane log file is named contrail-vrouter-dpdk.log. Depending on the version or installation methods, it can be located in different folders or even with a totally different name. For example:

This log file contains lots of good information that is helpful to understand the current running status. Understanding the log messages are important during troubleshooting process.

Having understood the lcore mapping basics, let’s start a test with some traffic flowing.

Let’s look at a multiple queue example.

Based on the previous setup, this time we added one more queue in tap interface of VM gen and then collect the core interface mapping:

Here is the table view of these mapping:

So most items remains the same, except we have one more queque added on tap and the vRouter interface to which it attaches, correspondingly, one core is allocated to serve this new queue. Before this new queue was created we already know that each of our lcores is serving same amount of queues, therefore as a "tie breaker", which we’ve mentioned when we introduce dpdkinfo -c previously, the first forwarding lcore, lcore#10 with our notation, is allocated for the new queue.

Let’s check the traffic distribution between lcores with multiple queues on VM tap interface:

Now, since we have multiple queues on both VM tap interface and the fabric interface. Traffic on all lcores are very well balanced. Please keep this in mind as an "ideal" traffic profile that we are expecting the vRouter to has. In production, we usually deal with more complicated vRouter lcore configurations and traffic profiles, so the lcore balancing may be appear as perfect as what we are seeing in lab environment, but at least you have a good baseline in your mind and knows what to look when the result is far worse than expected.

TODO: